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1.0  Executive  Summary 


1.1  Background 

Recent  Department  of  Defense  (DoD)  directives  call  for  joint  medical 
surveillance.  Joint  Vision  2010-2020  states  the  goals  of  Information  Superiority 
and  Full  Spectrum  Dominance.  In  addition,  the  emphasis  on  early  detection  of 
chemical  and  biological  attacks  makes  it  imperative  to  conduct  rigorous  testing 
and  evaluation  (T&E)  of  medical  informatics  technologies  under  development  to 
enhance  joint  force  protection. 

The  Medical  Data  Surveillance  System  (MDSS)  is  a  Web-based  automated 
surveillance  and  data  analysis  tool  intended  to  integrate  medical  information  for 
surveillance  of  deployed  forces  and  patient  populations  in  the  United  States.  The 
present  study  evaluated  MDSS  version  3.1,  focusing  on  its  functioning  and  utility 
for  end  users  at  Navy  and  Marine  Corps  MTFs. 

1.2  Method 

MDSS  was  installed  at  Naval  Medical  Center,  San  Diego  (NMCSD),  Naval 
Hospital  Camp  Pendleton  (NHCP),  121st  Evacuation  Hospital  in  Korea,  and  Naval 
Hospital  Okinawa  (NHO).  ICD-9  codes  from  outpatient  encounters  were  used  as 
the  primary  data  source  for  MDSS. 

The  T&E  plan  included  3  phases.  The  first  two  phases  consisted  of  administering 
surveys  to  medical  users  at  two  key  junctures:  immediately  following  their  initial 
training  on  MDSS  (n  =  20);  and  following  extended  use  3  months  later,  after  they 
had  an  opportunity  to  use  the  product  routinely  on  the  job  (n  =  10).  The  brief 
surveys  and  follow-up  interviews  measured  user  satisfaction  and  utility  of  the 
product  for  their  jobs. 

The  final  phase,  validation  testing,  consisted  of  direct  observation  of  MDSS 
performance  during  computer-based  tests  conducted  by  independent  subject 
matter  experts  (SMEs)  (n  =  3)  who  executed  a  representative  list  of  tasks  using 
MDSS.  A  standard  database  of  real  patient  encounters  was  used  for  data  input. 
This  validation  test  determined  whether  MDSS  performed  basic  functions  as 
intended  by  its  developers.  The  T&E  team  and  the  developer  agreed  on  a  set  of 
administrative,  data  analysis  and  surveillance  tasks  which  served  to  define 
operationally  these  intended  functions. 
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1.3  Results 


Medical  personnel  who  responded  to  the  posttraining  survey  were  active  duty,  and 
most  (65%)  had  positions  related  to  preventive  medicine,  epidemiology,  or 
environmental  health.  Respondents  provided  consistently  positive  ratings  of  the 
MDSS  product  and  the  training  process. 

A  subset  of  these  medical  personnel  responded  to  an  extended  use  survey  about  3 
months  later.  These  respondents  were  active  duty,  most  (6  of  10)  with  jobs 
directly  related  to  preventive  medicine  or  environmental  health.  Half  (3  of  6)  of 
these  latter  respondents  used  MDSS  regularly,  at  least  a  few  times  a  week. 

Overall,  data  from  all  10  respondents  to  the  extended  use  survey  showed  positive 
ratings  of  the  MDSS  features;  they  favored  the  data  analysis  tools  and  reports 
particularly  for  retrospective  analyses.  However,  these  ratings  were  not  as 
consistent  and  strongly  positive  as  the  posttraining  data.  Respondents  agreed  that 
MDSS  was  better  than  previous  reporting  methods,  but  they  were  undecided  as  to 
whether  it  helped  them  to  do  their  jobs.  Data  input  problems  prevented  regular 
use  and  evaluation  of  the  key  MDSS  feature,  the  alert  matrix.  There  was  no  clear 
example  of  MDSS  as  the  primary  detection  tool  for  disease  outbreak. 

Validation  tests  showed  that  MDSS  performed  its  basic  functions  as  intended  by 
its  developers.  SMEs  raised  issues  concerning  fine-tuning  of  the  system  to  meet 
their  needs  and  increase  usability.  They  commented  that  graphs  and  reports  were 
valuable,  but  some  report  categories  were  not  clinically  functional.  There  was 
concern  that  the  MDSS  alert  system  produced  too  many  “false  alarms.” 

1.4  Conclusions 

The  results  indicate  that  MDSS  version  3.1  functioned  as  intended  by  developers 
under  controlled  conditions  such  as  the  validation  tests  and  training  presentations. 
However,  after  extended  use  at  MTFs,  users  reported  reservations  about  whether 
the  present  version  of  the  product  helped  them  to  do  their  jobs.  Users  still  rated 
the  product  favorably  in  general,  with  some  specific  ideas  about  how  to  fine-tune 
MDSS.  In  general,  users  whose  jobs  were  directly  related  to  disease  surveillance 
and/or  preventive  medicine  favored  the  concept  behind  MDSS  but  did  not  believe 
it  was  ready  to  use  routinely  on  the  job. 

Though  users  were  satisfied  with  data  analysis  tools  and  graphics,  which  are 
critical  investigation  capabilities  of  MDSS,  the  system’s  alerting  capability  was  a 
particular  source  of  concern.  A  lack  of  timely  and  reliable  data  input  to  MDSS 
prevented  its  key  feature,  the  alert  matrix,  from  being  tested.  Only  one  user  was 
satisfied  as  to  the  reliability  of  data  input,  but  reported  that  the  alert  matrix 
produced  too  many  alerts.  Thus,  further  evaluation  of  this  alerting  capability  will 
be  necessary. 
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The  pattern  of  data  seen  from  several  sources,  including  validation  tests, 
numerical  ratings  and  comments  from  posttraining  and  extended  use  surveys, 
supports  the  validity  of  the  present  T&E  plan  and  its  measurement  instruments. 
Various  suggestions  for  improvement  of  the  MDSS  product  for  ease  of  use  and 
utility  are  listed  in  the  results  and  conclusions  sections  of  this  report.  These 
include  designing  outbreak  alerts  based  on  clinical  functional  syndrome  groups, 
adding  input  modules  such  as  pharmacy  data,  easier  access  to  Excel  pivot  tables, 
and  development  of  standard  operating  procedures  for  MDSS.  Recommended 
next  steps  for  T&E  of  this  product  are  presented  in  the  conclusions  section  of  this 
report. 
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2.0  Introduction 


Although  MDSS  and  associated  systems  have  been  evaluated  within  the  Joint  Medical 
Operations-Telemedicine  (JMO-T)  Advanced  Concepts  Technology  Demonstrations 
(ACTD)  for  the  objectives  of  Joint  Operations,  additional  testing  is  needed  to  validate 
consistent  performance  and  assess  the  system’s  capability  to  meet  Navy  and  Marine 
Corps  services  specific  requirements.1,2 

The  purpose  of  the  present  research  was  to  conduct  independent  T&E  of  MDSS  version 

3.1  during  an  extended  period  at  First  Marine  Expeditionary  Forces  (I  MEF)  units  at 
Camp  Pendleton  and  to  Command  and  Control  (C2)  activities  at  NHCP,  NHO,  NMCSD, 
and  the  121st  Evacuation  Hospital  in  Korea.  The  present  report  is  a  condensed  version  of 
a  comprehensive  final  report  prepared  by  the  contractor  and  available  upon  request.3 

2.1  Background 

MDSS  is  a  promising  medical  informatics  technology  approaching  maturity. 
Developed  by  the  Concept  Exploration  Laboratory  (CXL)  and  Naval  Health 
Research  Center  (NHRC)  in  San  Diego,  MDSS  promises  to  enhance  medical 
support  for  the  Navy  and  Marine  Corps.1 

The  development  of  MDSS  is  consistent  with  the  following  DoD  directives, 
which  identified  medical  surveillance  as  important  for  maintaining  force 
readiness: 

•  DoD  Directive  6490.2,  Joint  Medical  Surveillance,  30  August  1997 

•  DoD  Instruction  6490.3,  Implementation  and  Application  of  Joint  Medical 
Surveillance  for  Deployments,  07  August  1997. 

In  addition,  MDSS  addresses  the  current  Joint  Vision  2010-2020,  which  states  the 
goals  of  Information  Superiority  and  Full  Spectrum  Dominance.  The  recent  and 
continuing  emphasis  on  detection  and  prevention  of  chemical  and  biological 
attacks  has  made  research  and  development  of  medical  informatics  technologies  a 
pressing  need  for  deployed  forces. 

Previous  demonstrations  (JMO-T  ACTD  Cobra  Gold  ’01,  Kernel  Blitz  ’01)2  and 
discussions  among  product  developers  and  medical  users  at  sites  such  as  Korea 
have  done  much  to  advance  the  product.  Of  all  the  JMO-T  component 
applications,  MDSS  received  the  highest  marks  for  military  utility.  However,  no 
T&E  work  by  a  research  group  independent  of  the  developer  has  been  conducted 
emphasizing  Navy  applications  of  MDSS.  Data  collection  during  previous 
military  exercises  and  demonstrations  was  limited  to  2  or  3  weeks;  therefore, 
issues  of  utility  or  user  satisfaction  during  longer-term  use  must  still  be  evaluated. 

Feedback  from  users  and  developer  modifications  emerging  from  use  at  Branch 
Medical  Clinic  Chinhae  in  Korea  have  led  to  successive  versions  of  MDSS.  The 
focus  of  the  present  evaluation  will  be  MDSS  version  3.1. 
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NHRC  command  believed  that  MDSS  would  benefit  from  systematic  evaluation 
by  a  T&E  team  with  its  own  principal  investigator  independent  of  the  product 
development  team.4  Such  an  evaluation  would  also  allow  Navy  command  a  base 
from  which  to  make  decisions  on  the  suitability  of  the  product  for  future 
deployment.  Thus,  the  NKRC  telemedicine  and  T&E  group,  which  has  previous 
experience  in  the  evaluation  of  military  medical  technologies5'9,  was  tasked  to 
develop  and  execute  a  T&E  plan. 

2.2  Objectives 

The  present  evaluation  assesses  the  capability  of  MDSS  to  meet  the  needs  of 
Navy  and  Marine  Corps  medical  users  and  validates  whether  the  system  functions 
as  intended.  Specifically,  MDSS  was  evaluated  under  the  following  conditions: 

•  Long-term  use  at  Navy  and  Marine  Corps  settings 

•  Theatre-type  simulation  with  multiple  nodes  or  sites 

•  Overseas  MTF  sites. 

2.3  Description  of  MDSS 

MDSS  is  a  Web-based  information  system  that  allows  real-time  medical  threat 
assessment  for  deployed  forces.  The  software  facilitates  response  to  medical 
threats  by  providing  the  preventive  medicine  officer  (PMO),  epidemiologist,  or 
Commander-in-Chief  (CINC)  Surgeon  with  automated  tools  to  assist  in  the 
process  of  investigating,  identifying,  and  reporting  significant  medical  events. 

MDSS  automatically  searches  patient  data  using  ICD-9  codes  to  provide 
epidemiologists  with  tools  for  early  detection  of  disease  outbreaks  and  chemical 
attacks.  The  key  feature  of  MDSS  is  advanced  Dynamic  Change-Point  Detection 
(DCD)  analysis,  which  is  designed  to  allow  early  detection  of  illness  trends  and 
disease  outbreaks.  MDSS  uses  a  set  of  dynamic  change  point  and  signal  detection 
algorithms  to  identify  the  start  and  end  points  of  medical  events,  trends,  and  shifts 
within  routinely  collected  data.  The  system  is  able  to  identify  incidence  spikes 
using  relatively  small  data  sets  and  can  calculate  baselines  using  a  week’s  worth 
of  data  or  less  from  an  MTF.  MDSS  can  also  generate  standard  reports  such  as 
Disease  and  Non-Battle  Injury  (DNBI)  reports. 


2.4  Functions  to  Be  Tested 

During  T&E,  medical  personnel  had  interactive  access  to  MDSS  with  a  standard 
Web  browser  interface.  Through  this  interface,  they  were  able  to  perform  a 
variety  of  analyses  and  reports.3 

The  following  design  functions  of  MDSS  were  evaluated  for  user  satisfaction  and 
utility  of  MDSS  to  enhance  medical  support  for  the  Navy  and  Marine  Corps: 
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•  Preventive  medicine,  such  as  timely  surveillance  and  detection  of  disease 
threats 

•  Analysis  of  clinical  encounters 

•  Generation  of  standard  reports  such  as  DNBI 

•  Administrative  functions  such  as  controlling  access  to  users  and  the  MTFs 
a  user  can  view  during  surveillance. 


2.5  Installation  Plan 

T&E  personnel  determined  that  the  most  practical  method  for  obtaining  the  data 
needed  for  detection  of  emerging  medical  threats  such  as  disease  outbreaks  was  to 
access  records  of  outpatient  encounters.  Therefore,  the  data  needed  for  MDSS  in 
a  Navy  or  Marine  Corps  setting  were  drawn  from  real  and  ongoing  patient 
encounters  captured  by  the  SNAP  Automated  Medical  System  (SAMS)  and  for 
the  Ambulatory  Data  System  (ADS)  or  Composite  Health  Care  System  (CHCS) 
within  the  Pacific  Area  Of  Responsibility.  MDSS  version  3.1  was  installed  at  the 
sites  listed  above  and  accessed  data  directly  from  these  MTFs  as  available. 

The  present  evaluation  assesses  MDSS  performance  at  multiple  sites  to  simulate  a 
theatre  scenario.  Users  were  able  to  view  and  analyze  medical  data  exchanged 
between  several  MTFs  in  a  geographic  region  of  operations  such  as  NMCSD  and 
NHCP  and  their  associated  clinics. 

Conceptually,  the  deployment  of  MDSS  will  result  in  a  system  of  data  collection 
and  analysis  that  supports  the  functional  goals  mentioned  (Figure  1). 


Figure  1.  Anticipated  Configuration 
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2.6  Three  Phases  of  MDSS  T&E 


The  evaluation  team  assessed  the  reactions  of  medical  users  to  MDSS  at  the 
deployed  sites  at  two  key  junctures: 

•  The  time  of  their  initial  training  with  the  product 

•  Three  months  later,  when  they  had  used  the  product  routinely  on  the  job 
for  an  extended  period. 

In  addition,  experts  were  asked  to  use  MDSS  to  interpret  a  test  database 
consisting  of  real  patient  encounters  to  validate  whether  MDSS  functions  as 
intended  by  product  developers. 

2.6.1  Posttraining  Evaluation 

After  MDSS  was  installed  and  functional  at  each  of  the  deployed  sites,  the 
developers  provided  formal  training  which  consisted  of  2  days  of 
computer-based  instruction.  The  T&E  team  then  administered 
questionnaires  to  assess  trainees’  reactions  to  the  product. 

2.6.2  Extended  Use  MDSS  Evaluation 

For  3  months  after  they  had  been  trained,  the  medical  users  at  each  site 
were  asked  to  use  MDSS  as  a  routine  part  of  their  jobs.  After  this  period, 
the  T&E  team  administered  a  questionnaire  to  assess  the  users’  reactions 
to  MDSS.  The  T&E  team  followed  up  on  responses  to  this  questionnaire 
with  informal  individual  interviews  at  NMCSD  and  NHCP. 

2.6.3  Validation  Testing 

The  last  phase  of  the  present  research  consisted  of  controlled  tests  of  the 
system’s  functions.  MDSS  SMEs  were  asked  to  complete  a  set  of  tasks 
representative  of  important  system  functions  (Appendix  C)  using 
retrospective  patient  data. 
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3.0  Method 


This  evaluation  tested  whether  MDSS  met  design  objectives  intended  by  developers. 

This  T&E  plan  also  assessed  whether  MDSS  was  an  improvement  over  current  methods 
for  medical  surveillance.  The  conceptual  factors  addressed  included  system  reliability 
and  medical  user  satisfaction. 

The  sites  for  extended  use  evaluations  consisted  of  Navy  hospitals  where  MDSS  and 
associated  technologies  were  deployed  for  between  2  and  3  months,  with  some  variation 
between  sites  due  to  logistical  problems.  Sites  included  NMCSD,  NHCP,  NHO  and  the 
121st  Evacuation  Hospital  in  Korea. 

3.1  Data  Sources 

Two  data  sources  provided  information  to  meet  the  T&E  objectives: 

•  Structured  interviews  with  medical  users  of  MDSS 

•  A  validation  study  of  the  intended  functions  of  MDSS  as  agreed  on  by  the 
developer  and  the  T&E  team  at  the  study’s  onset. 

3.2  Structured  Interviews  With  Medical  Users  of  MDSS 

Two  questionnaires  were  administered: 

•  A  posttraining  survey  administered  immediately  after  MDSS  training  for 
new  users  (N  =  20)  (Appendix  A). 

•  An  extended  use  survey  administered  approximately  3  months  following 
training,  after  users  had  the  opportunity  to  work  with  the  system  as  a 
routine  part  of  their  jobs  (N  =  10)  (Appendix  B). 

Individual  interviews  with  medical  users  were  conducted  as  they  were  available  to 
follow  up  on  their  questionnaire  responses.  These  interviews  included  assessment 
of  value  added  to  normal  business  practices. 


3.2.1  Development  of  Structured  Interviews 

The  questionnaires  and  interview  format  were  based  on  several  sources  of 
information,  including: 

•  MDSS  training  and  user  guides 

•  Preliminary  information  such  as  After-Action  Reports,  surveys  and 
survey  approaches  provided  by  JMO-T  T&E  groups  from  previous 
DoD  exercises  and  demonstrations  (e.g.,  Thailand,  May  2002) 

•  Consultations  with  individuals  who  participated  in  the  evaluation 
of  these  exercises 


8 


•  Informal  interviews  with  MDSS  developers  and  relevant  NHRC 
personnel,  focusing  on  the  desired  outcomes  of  MDSS  training 
procedures  and  of  deploying  the  system  itself 

The  T&E  group  used  brief,  open-ended  questions  to  encourage  the 
relatively  few  respondents  to  detail  their  reactions  through  written  and 
verbal  responses.  Multiple  choice  questions  with  a  limited  set  of 
predetermined  outcomes  might  have  imposed  the  researchers’  or 
developers’  ideas  on  users.10  Each  item  in  the  survey  also  included  a 
5-point  Likert  scale  of  agreement  or  disagreement. 

3.2.2  Posttraining  Questionnaire 

After  each  training  session,  a  T&E  team  member  invited  trainees  to 
complete  a  voluntary,  confidential  survey  (Appendix  B).  On  this 
posttraining  questionnaire,  users  were  asked  to  describe  their  job 
functions,  reactions  to  training,  and  initial  reactions  to  MDSS.  Users 
completed  the  questionnaires  in  approximately  10  to  15  minutes  and 
returned  the  questionnaires  directly  to  T&E  team  members  or  returned 
them  by  mail.  No  follow-up  interviews  were  conducted  based  on  the 
posttraining  questionnaire. 

Each  respondent  had  the  opportunity  to  write  comments  after  completing 
the  rating  scale  for  each  item  in  the  posttraining  survey.  These  comment 
responses  were  assigned  to  categories  that  emerged  from  the  initial  review 
of  all  comments  in  the  surveys.  Each  comment  was  assigned  to  only  one 
category.  Users  could  contribute  zero,  one,  or  multiple  comments  for  each 
survey  item,  and  comments  for  all  items  across  respondents  were  pooled 
for  this  analysis.  Every  comment  was  assigned  to  a  category,  even  if  the 
respondent  had  made  a  similar  comment  in  a  previous  question.  The 
assignment  of  the  comment  categories  proved  to  be  100%  reliable  when 
two  researchers  assigned  comments  to  categories  independently  in  both 
the  post-training  survey  and  the  extended  use  survey. 

Comment  analysis  was  conducted  separately  for  comments  from  the 
posttraining  surveys  and  comments  from  the  extended  use  surveys. 

Comment  categories  ranged  from  “ease  of  use”  to  “aspects  of  data 
reports.”  “Concerns  with  data  reliability”  was  a  category  that  appeared  in 
both  the  posttraining  and  extended  use  surveys.  The  following  are 
examples  of  comments  from  this  latter  category: 

•  “Though  the  connection  between  the  data  repository  and  MDSS 
was  established,  the  inadequacies  in  collection  and  reporting  of 
data  decreased  ability  to  detect  significant  increases  in  disease 
trends.  Data  not  yet  reliable.” 
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•  “The  data  input  problem,  namely  that  clinics  report  only  at  the  end 
of  the  month,  would  affect  immediate  capabilities  of  any  system 
like  MDSS.” 

3.2.3  Extended  Use  Questionnaire 

When  users  had  the  opportunity  to  apply  MDSS  in  their  jobs  for  3  months 
after  training,  an  extended  use  questionnaire  (Appendix  B)  was 
administered.  Nine  of  the  ten  respondents  had  attended  one  of  the  formal 
MDSS  training  sessions.  The  remaining  respondent  had  had  access  to 
MDSS  for  approximately  one  year  prior  to  formal  training  procedures  and 
had  learned  through  self-study  and  informal  discussion  with  developers. 

This  voluntary,  confidential  questionnaire  obtained  the  following 
information: 

•  Medical  users’  backgrounds  (e.g.,  PMO) 

•  Extent  of  MDSS  use  (e.g.,  daily,  weekly) 

•  How  users  employed  MDSS  in  their  jobs 

•  User  reactions  to  general  and  specific  design  features  and  functions 
of  MDSS  (e.g.,  medical  surveillance,  drill-down  capabilities) 

•  Advantages  and  value  added  to  job  performance  by  MDSS. 

The  T&E  team  immediately  reviewed  each  completed  questionnaire  and 
scheduled  telephone  or  in-person  interviews  to  follow  up  on  responses  to 
critical  items. 


3.3  Validation  Study  of  the  Intended  Functions  of  MDSS 

A  validation  test  was  designed  to  document  whether  MDSS  executed  its  basic 
functions.  A  list  of  tasks  representative  of  MDSS  functions  was  prepared  by  the 
T&E  team  and  agreed  on  by  the  product  developer  (Appendix  C).  Under 
controlled  conditions,  SMEs  (N  =  3)  used  MDSS  to  perform  these  tasks.  The 
SMEs  were  experienced  medical  users  of  at  least  two  months  from  two  of  the 
MDSS-deployed  sites  (NMCSD  and  NHCP)  and  a  third  site  in  San  Diego. 

3.3.1  Test  Procedure 

Validation  T&E  of  MDSS  was  performed  in  two  phases: 

1 .  A  test  database  was  assembled  based  on  retrospective  patient 
encounters  at  NMCSD.  The  database  contained  approximately 
30,000  patient  records  from  the  month  of  September  2002.  The 
test  database  and  MDSS  version  3.1  were  installed  on  a  5.84  GB 
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Dell  Latitude  notebook  computer.  This  allowed  the  tests  to  be 
conducted  without  Internet  connection  to  MDSS  online. 

2.  The  tasks  were  representative  of  various  user  roles,  such  as 
administrative  functions  and  medical  surveillance  functions,  and 
were  completed  in  approximately  30  minutes.  The  SMEs  recorded 
whether  or  not  each  task  was  performed  successfully.  Success  was 
defined  as  MDSS’s  completion  of  assigned  tasks,  such  as  those 
listed  below,  without  a  system  crash  or  substantial  delay. 


•  Generating  reports  such  as  DNBI 

•  Detecting  trends,  patterns,  and  threats  with  DCD  analysis 

•  Correlations  of  outbreaks  to  patient  units  and  locations 
(i.e.,  drill-down  investigation) 

•  Selecting  a  population  at  risk  (PAR) 

•  Data  analysis  features  (e.g.,  pivot  tables). 

In  practice,  success  was  defined  by  the  judgment  of  SME  and  T&E 
observers,  who  were  in  agreement  on  success/failure  on  all  tasks  run. 
Some  usability  issues  were  noted  by  SMEs  and  were  recorded  on  the  task 
log  sheet  (Appendix  C). 

MDSS  version  3.1  had  previously  been  tested  for  accuracy  in  generating 
counts  for  various  patient  conditions  and  reports  such  as  the  DNBI  by 
contractor.1 1  The  present  authors  also  conducted  informal  tests  that 
confirmed  that  MDSS  counts  for  a  given  set  of  patient  records  matched 
those  counts  calculated  independently  by  Excel  software  for  the  same  set 
of  patient  records. 
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4.0 


Results 


4.1  Demographics  of  Trainees 

Table  1  summarizes  relevant  demographics  for  the  20  trainees.  Though  duty 
status  and  service  were  not  recorded  on  the  survey,  informal  observations 
indicated  that  the  trainees  were  almost  all  active-duty  Navy  or  Marine  Corps 
personnel.  Most  (13  of  20)  had  positions  in  preventive  medicine,  epidemiology, 
or  environmental  health.  Others  included  a  corpsman,  a  radiation  safety  specialist 
and  an  infection  control  nurse.  Almost  two  thirds  of  the  trainees  (13  of  20)  were 
associated  with  NHCP,  with  the  remaining  7  individuals  associated  with  NMCSD 
(n  =  3)  or  NHO  (n  =  4). 


Table  1.  Demographics  of  Respondents  to  Posttraining  Survey 


N 

% 

Gender 

Male 

12 

60 

Female 

8 

40 

Location 

NH  Camp  Pendleton 

13 

65 

NMC  San  Diego 

3 

15 

NH  Okinawa 

4 

20 

Position 

Preventive  Medicine 

7 

35 

Corpsman 

2 

15 

Environmental  Health  Officer 

4 

20 

Epidemiologist 

2 

10 

Other 

5 

20 

4.1.1  Numerical  Ratings  by  Trainees 

Figure  2  shows  that  respondents  provided  consistently  positive  mean 
(+/-  V2  standard  deviation)  ratings  of  the  MDSS  product  and  the  training 
process.  Scores  closer  to  5  indicated  more  agreement  with  each  statement. 
The  mean  ratings  for  all  5  questions  were  at  least  4.0,  indicating 
agreement  with  the  statements  in  the  survey  such  as  “I  am  prepared  to  use 
MDSS,”  and  “I  would  recommend  MDSS  to  my  command.” 
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Figure  2.  Training  Questions 
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4.1.2  Usefulness  Ratings  of  MDSS  Features 

Figure  3  shows  that  respondents  provided  consistently  positive  ratings  of 
MDSS  features  such  as  the  alert  matrix  and  drill-down  capabilities  when 
asked  to  judge  their  usefulness  based  on  the  training  presentation  they  had 
just  completed.  The  mean  (+/-  lA  standard  deviation)  ratings  for  all  7 
features  listed  were  at  least  4.0  on  the  rating  scale,  indicating  that  the 
respondents  judged  them  to  be  useful. 


Figure  3.  Usefulness  Ratings 
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4.1.3  User  Comments  on  Survey  Questions 

Figure  4  shows  the  percentage  of  all  comments  (N  =  47)  for  various 
categories  of  information.  All  20  respondents  contributed  at  least  one 
comment.  The  most  prevalent  comment  was  that  MDSS  was  a  good 
system  (25%).  With  regard  to  the  training  process,  respondents  often 
commented  that  they  would  have  to  use  the  system  more  (21%)  and  that 
the  training  was  well  run  (17%).  Informal  observations  often  indicated 
that  users  were  anxious  to  test  the  system  capabilities  based  on  the 
potential  that  they  had  seen  during  training. 

Relatively  few  negative  comments  were  made  about  unorganized  training 
(11%)  and  concerns  about  data  quality,  data  sources,  and  relevance  to  the 
respondent’s  command.  The  T&E  team’s  specific  recommendations  to 
address  training  improvements  were  presented  above. 
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Figure  4,  Comment  Categories 


4.2  Extended  Use  Survey  Data 
4.2.1  Demographics 

Ten  individuals  responded  to  the  extended  use  survey.  Nine  of  the 
respondents  were  tracked  from  their  attendance  at  MDSS  introductory 
training  sessions.  There  was  one  exception,  a  long-term  user  overseas 
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who  learned  MDSS  prior  to  the  initiation  of  the  formal  MDSS  training 
sessions. 

Table  2  shows  that  these  individuals  were  mostly  active-duty  Navy 
personnel,  primarily  males  from  the  San  Diego  or  Camp  Pendleton  sites. 
Most  respondents  had  job  positions  relating  to  preventive  medicine  or 
environmental  health.  They  also  noted  training  in  epidemiology  and 
preventive  medicine,  with  substantial  experience  and  duties  related  to 
disease  surveillance. 


Table  2.  Demographics  of  Respondents  to  Extended  Use  Survey 


N 

% 

Gender 

Male 

9 

90 

Female 

1 

10 

Duty  Status 

Active 

10 

100 

Other  (civilian,  retired) 

0 

0 

Service 

Navy 

9 

90 

Army 

1 

10 

Location 

121st  Korea 

1 

10 

NH  Camp  Pendleton 

6 

60 

NMC  San  Diego 

2 

20 

Branch  Medical  Clinic 
Barstow 

1 

10 

Position 

Preventive  Medicine  Officer 

3 

30 

Corpsman 

4 

40 

Environmental  Health 

Officer 

3 

30 

4.2.2  MDSS  Usage  Frequency 

The  respondents  reported  how  frequently  they  used  the  system  and 
described  the  nature  of  their  usage.  All  individuals  had  formal  or  informal 
introductions  and  training  in  MDSS.  Almost  all  had  at  least  intermittent 
access  to  the  Web-based  application  during  the  2-  to  3 -month  study 
period. 
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As  seen  in  Figure  5  (n  =  10),  7  of  the  respondents  (70%)  used  MDSS 
infrequently;  they  said  that  they  rarely  used  MDSS  or  used  it  a  few  times  a 
month.  Most  of  these  individuals  did  not  have  positions  directly  related  to 
surveillance  (i.e.,  4  of  these  7  respondents  were  corpsmen).  Informal 
feedback  from  respondents  and  developers  confirmed  that  MDSS 
connections  to  MTFs  were  more  reliable  than  connections  to  outlying 
clinics  where  these  corpsmen  worked. 

The  remaining  3  individuals  reported  using  MDSS  regularly,  a  few  times  a 
week  or  daily.  These  3  individuals  worked  in  positions  directly  related  to 
surveillance,  with  training  in  preventive  medicine  and/or  epidemiology. 
These  respondents  worked  at  MTFs,  2  in  the  United  States  and  1  overseas, 
where  MDSS  had  been  operational  for  over  a  year. 

In  summary,  4  respondents  were  corpsmen,  and  MDSS  was  not  very 
relevant  for  their  jobs.  MDSS  was  most  relevant  for  the  jobs  of  6  of  the 
respondents  in  this  extended  use  survey  sample.  Fifty  percent  of  these  6 
individuals  (3  of  6)  reported  using  MDSS  regularly  in  their  jobs  during  the 
study  period. 
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Figure  5.  How  Often  MDSS  Presently  Used 

4.2.3  Numerical  Ratings  for  Usefulness  and  Satisfaction 

After  questions  relating  to  demographics  and  frequency  of  use  of  MDSS, 
the  remaining  questions  in  the  extended  use  survey  focused  on  measuring 
user  reactions  to  the  use  of  MDSS,  including  numerical  ratings  and  coding 
of  written  comments. 

Figure  6  summarizes  mean  ratings  (and  range  of  scores,  e.g.,  4-5)  for 
questions  about  the  utility  of  MDSS  information.  Overall,  users  provided 
consistently  favorable  ratings  of  MDSS  for  its  medical  threat  information 
and  usability.  The  mean  and  median  scores  (medians  not  shown)  for  1 1  of 
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12  questions  were  on  the  upper  end  of  the  5-point  scale,  indicating 
positive  reactions  to  the  MDSS  product  (e.g.,  MDSS  provided  useful 
information;  MDSS  was  easy  to  navigate;  MDSS  information  was 
presented  in  a  useful  format).  Scores  closer  to  5.0  indicated  more 
agreement  with  the  survey  statements.  The  extended  use  survey  is 
attached  in  Appendix  B  for  reference.  Samples  sizes  across  survey  items 
varied  between  3  and  6  respondents  because  not  all  survey  items  were 
relevant  for  all  respondents  (e.g.,  respondents  marked  “not  observed”). 
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Figure  6.  Questions  (Range  of  scores  shown  above  each  bar) 

The  figure  shows  variability  between  questions  in  the  average  ratings. 
Several  critical  questions  deserve  emphasis  here: 

Users  were  asked  if  the  quality  of  MDSS  medical  threat  information  was  better  than  that 
supplied  by  previous  reporting  methods  (Q  #6).  This  question  received  the  most 
favorable  rating  (mean  =  4.3,  or  between  strongly  agree  and  agree). 

Users  were  asked  if  MDSS  was  easy  to  use  (Q  #2).  This  question  received  a  relatively 
low  rating  (mean  score  =  3.5,  or  between  agree  and  undecided). 

When  asked  if  MDSS  helped  them  to  do  their  jobs,  the  users  were  undecided,  neither 
agreeing  nor  disagreeing  with  the  statement  (average  =  3.0).  This  question  received  the 
lowest  rating  of  all  the  questions. 

Finally,  users  agreed  that  the  introductory  training  and  CD  reference  guide  (provided 
several  months  before  the  training)  were  helpful  introductions  to  MDSS. 

Because  of  small  sample  sizes,  it  is  difficult  to  draw  statistically  reliable 
conclusions  from  comparing  individual  questions.  These  contrasts  will  be 
considered  later  in  the  discussion  with  other  information  -  such  as  ratings 


of  individual  features,  user  comments  and  validation  tests  -  to  determine  if 
a  consistent  pattern  of  results  emerges. 


4.2.4  Numerical  Ratings  for  MDSS  Features 

Figure  7  summarizes  mean  (and  range  of  scores,  e.g.,  1-5)  user  ratings  for 
satisfaction  with  specific  MDSS  capabilities,  such  as  alert  matrix,  reports, 
and  DCD  capabilities.  The  mean  and  median  ratings  (medians  not  shown) 
indicated  that  users  were  satisfied  or  very  satisfied  with  most  of  the 
features,  such  as  DNBI  reports,  DCD,  drill-down  capability,  reduced  staff 
calendar,  and  graphics.  Some  comments  include:  “Easy  to  compare 
current  data  to  old  [data]”  (referring  to  DCD),  and  “Those  graphs  are 
really  easy  to  look  at  a  glance.” 


"D 

0) 

«= 

to 

Ip 

CO 

to 

b 

4) 

> 


4) 

Ui 

2 
m 

<  -o 
0) 
IT 
to 

13 

to 

to 


■o 

£ 

0) 

> 


Alert  matrix 


DNBI 

reports 


Dynamic 

change 

point 

detection 


Pivot 

tables 


Epi 

tools/drill 

down 


PAR  tables 


Calendar 
for  staff 
days 


Graphs 


Figure  7.  Satisfaction  with  Functions  (Range  of  scores  show  above  each  bar) 


Several  remaining  features  received  slightly  less  favorable  ratings  between 
undecided  and  satisfied,  such  as  the  alert  matrix,  pivot  tables  and  PAR 
tables.  One  comment  given  about  the  PAR  tables  was  that  they  “require 
data  input  on  my  part  but  unit  level  info  is  difficult  to  acquire  and 
maintain.”  A  comment  about  the  alert  matrix  stated  that  it  was  “not 
updated  in  real  time,”  meaning  that  there  is  a  lag  of  approximately  lday 
before  data  input  to  the  system.  (It  should  be  noted  that  the  alert  matrix 
was  dependent  on  timely  data  input,  so  the  updating  issue  is  not  an  MDSS 
design  issue.) 


18 


4.2.5  User  Comments 


Figure  8  summarizes  the  percentage  of  all  comments  (N  =  40)  for  various 
categories  of  information.  Five  respondents  provided  one  or  more 
comments.  For  this  analysis,  user  comments  for  all  questions  were  pooled 
and  coded  into  exclusive  categories. 


Figure  8.  Comment  Categories 


Several  results  are  apparent.  First,  there  were  more  negative  comments 
(60%,  coming  from  4  respondents)  than  positive  comments  (N  =  40%, 
coming  from  4  respondents).  The  positive  comments  mostly  reflected 
user  satisfaction  with  the  graphics  and  report-generating  capabilities  of 
MDSS.  The  negative  comments  mostly  reflected  data  input  problems, 
which  were  not  due  to  an  MDSS  design  issue  and  could  have  resulted 
from  network  overloading  or  slow  connections.  Negative  comments  also 
suggest  that  users  believed  that  reports  were  too  general  in  nature;  users 
also  seemed  to  want  to  see  diagnoses/complaints  more  directly  accessible 
as  an  MDSS  analytical  capability. 

4.3  Additional  User  Responses 

MDSS  users  offered  other  noteworthy  responses  in  the  extended  use  survey, 
including  opinions  on  the  most  useful  feature,  the  least  useful  feature,  and  the 
most  preferred  addition  to  MDSS. 


19 


4.3.1  Most  Useful  Feature 


Users  were  asked  what  was  the  most  useful  feature  of  MDSS  for  their 
jobs.  Four  users  offered  the  following  responses: 

•  Well-organized  data 

•  DNBI  index 

•  It  is  pretty  quick  since  CHCS  is  updated  daily 

•  Reports,  enhances  surveillance  capability  for  individual  clinics 
separately. 

4.3.2  Least  Useful  Feature 

Users  were  asked  what  was  the  least  useful  feature  of  MDSS  for  their  jobs. 
Three  users  offered  the  following  responses: 

•  Lack  of  data 

•  Dependent  on  CHCS  input 

•  Some  ICD-9  codes  are  off  base,  causing  the  alert  system  to  sound 
off. 

4.3.3  Most  Preferred  Addition 

Users  were  asked  what  one  thing  they  would  want  to  add  to  MDSS.  Four 
users  offered  the  following  responses: 

•  A  statistical  graph  based  on  a  local  map 

•  Trend  analysis 

•  Provide  e-mail  alert  to  notify  about  abnormal  alerts 

•  ER  application  and  pharmacy  data 

•  Top  ten  ICD-9  codes. 

Validation  Study  Results 

The  primary  purpose  of  implementing  a  validation  study  is  to  confirm  that  the  system 
works  as  intended.  The  intended  functions  were  defined  by  a  set  of  tasks  representative 
of  important  system  functions  (Appendix  C)  using  a  known  set  of  retrospective  outpatient 
data. 


5.1  Subject  Matter  Experts 

All  participating  SMEs  were  active-duty  Navy  officers.  One  was  a  PMO  with 
primary  training  in  epidemiology.  He  rarely  uses  MDSS  but  has  given 


demonstrations  of  MDSS  in  the  past,  including  a  demonstration  at  Kernel  Blitz  in 
2001 .  This  user  had  daily  on-the-job  experience  with  a  similar  system, 

ESSENCE,  for  surveillance. 

Another  SME  was  an  EHO  trained  in  epidemiology.  Primarily,  he  uses  MDSS 
retrospectively  to  send  reports  to  other  clinics.  His  main  interest  is  in 
surveillance.  Though  he  says  that  he  does  not  mind  tracking  down  alerts,  he  says 
it  is  just  a  matter  of  having  enough  time  to  do  it. 

The  third  SME  is  also  an  epidemiologist  and  uses  ESSENCE  every  day.  He  states 
that  until  now,  he  could  not  use  MDSS,  but  now  that  there  is  a  dedicated  server 
for  it,  he  will  most  likely  use  it  often. 

5.2  MDSS  Performance 

In  terms  of  overall  performance  with  reference  to  the  Validation  Task  List, 
Version  1,  July  31,  2002,  MDSS  displayed  the  following  as  intended  by 
developers: 

•  Trends  in  the  data,  through  graphs,  individual  patient  data,  and 
tables 

•  Alerts  of  disease  threats 

•  Alerts  of  potential  chemical/biological  attacks 

•  Report  generation  (e.g.,  DNBI,  Chemical  Biological  Radiological 
[CBR],  Reportable  Conditions,  Ill-Defined  Conditions,  Key 
Symptoms) 

•  Location  of  outbreaks 

•  Date  of  patient  visit 

•  Onset  and  offset  of  disease  trends  and  outbreaks. 

The  three  30-45  minute  validation  exercises  performed  support  this  conclusion. 

5.3  V alidation  T asks 

The  various  tasks  completed  during  the  validation  study  were  representative  of 
various  user  roles,  such  as  administrative  functions  (e.g.,  assigning  reduced  staff 
days),  medical  surveillance  functions  (e.g.,  viewing  background  alerts,  running 
DNBI  reports)  and  nonmedical  user  roles  (e.g.,  viewing  weekly  incidence  rates). 
(See  Appendix  C) 

The  validation  exercise  included  35  tasks:  10  tasks  under  the  administrative 
function,  19  surveillance  role  tasks,  and  6  nonmedical  role  tasks.  Not  all  SMEs 
completed  all  tasks  in  the  interest  of  time. 
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5.3.1  Administrative  Tasks 


Two  SMEs  completed  10  of  10  administrative  tasks  successfully,  whereas 
one  SME  skipped  this  first  section  because  of  time  restrictions  and  the  fact 
that  he  would  not  be  using  MDSS  administrative  functions. 

5.3.2  Surveillance  Tasks 

The  surveillance  role  was  the  longest  section,  and  all  3  SMEs  participated. 
(One  SME,  in  the  interest  of  time,  did  not  complete  all  of  the  tasks.)  Most 
of  the  testing  effort  was  concentrated  in  this  section  because  it  was  the 
most  relevant  for  the  SMEs. 

One  SME  tested  1 5  tasks,  and  all  1 5  were  successfully  completed.  One 
tested  1 6  tasks  and  completed  all  1 6  successfully,  and  one  tested  7  tasks 
and  completed  all  7  successfully.  The  reason  that  SMEs  were  asked  to  test 
only  a  subset  of  the  19  possible  tasks  was  to  shorten  the  session  to 
accommodate  the  SMEs’  schedules.  In  most  cases,  the  tasks  that  were 
intentionally  omitted  were  somewhat  redundant  with  previous  tasks. 

In  one  case,  2  SMEs  did  not  specify  which  MTF  they  wanted  to  look  at 
when  running  a  background  alert,  and  MDSS  successfully  alerted  them 
before  they  could  proceed.  During  testing,  several  issues  and  concerns 
were  raised  by  the  SMEs,  such  as  whether  rates  or  counts  were  supposed 
to  be  displayed,  all  of  which  are  addressed  in  the  next  section. 

Two  SMEs  agreed  that  MDSS  helped  or  would  help  them  in  their  jobs, 
though  a  third  considered  it  too  sensitive.  This  user  indicated  that  if 
modifications  were  made  in  the  ICD-9  grouping,  or  users  were  enabled  to 
break  out  bioterrorism  (BT)  reports/alerts,  he  would  be  motivated  to  use 
MDSS  more  frequently.  (Regarding  the  request  for  BT  reports,  the 
authors  of  this  evaluation  believe  that  further  training  with  MDSS,  and  its 
CBR  reports  in  particular,  could  make  the  system  more  helpful  for  this  and 
other  users.) 

In  summary,  these  validation  tests  showed  that  under  ideal  conditions 
(e.g.,  no  network  issues  or  running  reports  with  too  much  data),  MDSS 
functioned  as  intended. 

5.3.3  Nonmedical  Tasks 

Only  one  SME  participated  in  the  nonmedical  user  role,  because  the  tasks 
assigned  to  that  role  did  not  pertain  directly  to  the  SMEs’  jobs.  The 
individual  who  did  participate  in  the  nonmedical  role  section  skipped  two 
relatively  redundant  tasks  (because  he  had  completed  similar  tasks  in  the 
surveillance  section),  but  completed  3  of  the  remaining  4  tasks 
successfully.  One  task  (Reportable  Conditions  Summary  Report 
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generation)  was  flagged  for  review  when  it  appeared  that  the  display  was 
unintentionally  showing  the  entire  month  when  a  week  was  specified.  It 
was  later  determined  to  be  functioning  as  intended  by  developers,  and  all  4 
tasks  were  considered  to  completed  successfully.  This  issue  is  addressed 
below  in  the  Issues  and  Solutions  section. 

5.4  Issues  and  Solutions 

Numerous  issues  arose  during  the  validation  study  of  MDSS.  In  some  cases, 

solutions  were  proposed  to  address  these  issues. 

5.4.1  Surveillance  Analysis  Functions 

The  validation  study  led  to  a  question  regarding  the  surveillance  analysis 
functions.  Using  PAR  values,  the  surveillance  analysis  functions 
calculated  daily  and  monthly  incidence  rates  as  intended,  but  the  weekly 
incidence  rates  did  not  appear  to  be  calculated  correctly.  This  discrepancy 
was  determined  by  the  developers  and  confirmed  by  the  T&E  team  to  be 
due  to  a  missing  PAR  value;  since  the  PAR  value  was  not  entered,  the 
rates  were  not  being  calculated. 

5.4.2  False  Alarms 

Two  SMEs  voiced  concern  that  MDSS  was  too  sensitive,  that  there  were 
too  many  false  alarms.  The  literature  explains,  and  the  developers  agree, 
that  in  using  surveillance  systems,  you  need  to  know  what  you  are  looking 
for  to  understand  if  you  should  investigate  an  alert.12  The  third  SME 
agreed  with  this  perspective  and  gave  a  hypothetical  scenario:  if  you  are 
looking  at  a  place  with  a  very  small  population,  you  need  to  know  that  it  is 
a  very  small  population  because  small  numbers  will  cause  an  alert.  This 
user  explained  that  he  would  prefer  to  see  false  alarms  that  he  could  check 
and  use  as  a  guide,  rather  than  not  seeing  the  alert  at  all.  A  similar 
approach  was  advocated  in  the  development  of  decision  rules  for 
screening  for  possible  tuberculosis  cases  using  automated  health  data.13 

5.4.3  Reliability  of  Data  Input 

One  SME  was  concerned  about  the  reliability  of  data  input  from  SAMS  or 
CHCS  matching  MDSS  requirements.  He  noted  that  many  clinics  collect 
and  report  surveillance  data  as  aggregate  or  group  data  without  individual 
records  or  identifiers.  The  SME  wondered  whether  MDSS  should  be  able 
to  input  these  kinds  of  aggregate  or  group  data,  as  they  are  used  routinely 
by  clinics  and  reported  as  group  data  to  higher  levels  of  medical 
command. 
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5.4.4  Lack  of  Biochemical  Terrorism  Reports 

A  concern  arose  over  the  lack  of  reports  targeted  specifically  to 
biochemical  terrorism  outbreaks.  The  developers  explained  that  CBR 
reports  cover  those  kinds  of  alerts,  and  if  an  outbreak  were  to  occur,  it 
would  be  displayed  in  CBR  reports.  This  was  noted,  and  the  CBR  reports 
were  run  to  verify  that  biochemical  alerts  are  in  fact  displayed  within  the 
CBR  category.  When  this  was  verified,  it  was  concluded  that  the  user 
who  raised  the  concern  had  not  fully  understood  the  CBR  function. 

Further  training  in  this  area  may  serve  to  alleviate  the  user’s  concern. 
Where  possible,  mechanisms  should  be  developed  to  support  ongoing 
interaction  between  user  and  developer. 

5.4.5  Alert  Grouping 

An  SME  expressed  concern  that  the  alerts  are  not  grouped  by  syndrome 
and  therefore  might  not  be  effective  in  determining  a  serious  outbreak. 

The  developers  clarified  that  a  serious  outbreak  would  be  easily  detectable 
because  the  Symptoms  or  Ill-Defined  Conditions  reports  would  specify  it. 
The  developers  also  pointed  out  that  within  the  Key  Symptoms  alerts,  the 
larger  counts  in  each  symptom  grouping  are  listed  in  order  of  frequency 
and  group  together  at  the  top  of  the  report,  enabling  clinically  functional 
syndromes  to  be  determined.  Another  SME  agreed  with  this  conclusion. 
The  scenario  was  recreated  to  verify  this  conclusion,  and  it  was  found  that 
in  running  DNBI  reports  and  Key  Symptoms  alerts,  the  outbreaks  that 
occurred  were  grouped  together  by  occurrence  of  symptom.  Viewing 
these  two  reports  together,  a  user  could  determine  the  syndrome  that  was 
occurring.  This  sort  of  nuance  in  MDSS  usage  would  be  another  issue 
worth  emphasizing  during  training  and  ongoing  interaction  with  users  on 
the  job. 

5.4.6  Rate  per  Thousand 

There  was  a  question  whether  the  rate  per  thousand  was  a  rate  or  a  count. 
It  appeared  to  be  a  count  and  was  displaying  as  a  count  because  the  PAR 
number  had  not  been  entered. 

5.4.7  Links  to  Windows 

In  one  validation  exercise,  the  weekly  CBR  report  was  observed  to  have 
the  outpatient  counts  and  weekly  incident  rates  linked  to  different 
windows.  However,  the  weekly  rate  per  thousand  linked  to  the  same 
window  as  the  weekly  incident  rates.  An  SME  questioned  whether  these 
rates  should  link  to  the  same  window,  namely  the  graph  display.  It  was 
determined  not  to  be  a  fault  in  the  software,  which  functioned  as  intended 
by  developers.  The  outpatient  information  is  a  direct  means  to  view  a  list 
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of  individual  patient  information.  In  contrast,  the  incident  and  rate  per 
thousand  link  to  windows  that  display  graphs,  with  the  individual  patient 
information  available  in  an  Excel  file. 


5.5  Useful  Functions 

The  validation  tests  also  provided  insight  into  what  functions  of  MDSS  were 
considered  especially  useful.  These  include: 

•  DNBI  reports,  which  allow  for  double-checking  against  other 
sources  for  number  of  cases  and  to  identify  reports  that  are  not 
coming  from  other  sources. 

•  Dynamic  Change-Point  Detection  was  considered  by  one  SME  to 
be  the  most  useful  MDSS  tool. 

•  Reduced  staff  days  and  the  ability  to  enter  them. 

•  Having  both  counts  and  rates  displayed  is  useful. 

•  User-specified  analysis/control:  time  period,  key  symptoms. 

6.0  Recommendations 

Based  on  the  validation  tests,  surveys  and  interviews,  several  recommendations  for  the 
development  of  MDSS  can  be  offered. 

6.1  Excel/Pivot  Tables 

During  training  observations,  2  SMEs  and  some  trainees  considered  it 
cumbersome  to  move  data  into  Excel  and  use  pivot  tables.  This  fact  supports  the 
conclusion  that  this  function  should  be  modified  or  users  trained  differently.  To 
use  this  function  successfully,  users  are  expected  to  be  proficient  in  Microsoft 
Excel,  although  this  might  not  be  the  case.  It  is  possible  to  further  simplify  these 
features  by  adding  an  Excel  macro  or  a  graphical  user  interface  for  the  pivot  table 
functions. 

6.2  Additional  Data  Modules 

It  might  be  beneficial  to  investigate  adding  one  or  more  modules  to  MDSS.  The 
developers  of  ESSENCE  are  considering  expanding  their  system  by  adding 
pharmaceutical,  laboratory  and  radiological  exam  data  from  all  MTFs  currently 
capturing  this  information.  Accessing  these  data  might  provide  a  separate  but 
complimentary  and  confirmatory  model  to  the  system.10  Such  additional  data 
would  also  help  to  detect  diseases  that  occur  in  extremely  low  numbers,  such  as 
West  Nile  Virus  cases  or  the  few  anthrax  cases  that  occurred  through  the  U.S. 
Postal  system  in  October  2002. 
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6.3  ICD-9  Coding 


Although  MDSS  is  a  tool  and  should  not  be  substituted  for  human  interfacing  and 
decision  making,  certain  techniques  could  reduce  the  risk  of  questionable  data 
quality.  One  such  technique  is  the  grouping  together  of  ICD-9  codes  by  their 
clinical  utility.  This  might  decrease  the  impact  of  variation  in  ICD-9  coding 
practices.13  Coding  variations  were  a  concern  among  the  SMEs,  as  well  as  users. 
Currently,  for  example,  many  strep  patient  diagnoses  go  to  Ill-Defined  Conditions 
rather  than  respiratory,  and  this  might  decrease  or  obscure  the  detection  of  an 
outbreak.  Using  broad  syndromes  should  assist  in  trend  detection. 
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7.0  Conclusions  and  Recommendations 


This  study  consisted  of  the  T&E  of  MDSS  version  3.1,  a  Web-based  semi-automated 
disease  surveillance  system  intended  for  Navy  and  Marine  Corps  applications.  Survey 
data  gathered  after  training  presentations  and  extended  use  at  MTFs  and  observations 
during  controlled  validation  tests  indicated  that:  1)  the  product  functioned  as  intended  by 
developers,  2)  users  found  some  features  of  the  product  useful  for  their  jobs  but  overall 
they  were  not  convinced  it  was  ready  to  use  on  a  routine  basis  and,  3)  modifications  to 
the  product  may  enhance  its  utility  for  surveillance  work.  A  summary  of  evidence 
supporting  these  three  conclusions  is  presented  below. 

First,  the  data  from  the  validation  tests  demonstrated  that  MDSS  functioned  as  intended 
by  developers.  Independent  SMEs  were  able  to  execute  a  representative  set  of  functions 
successfully  without  significant  failures.  It  was  not  within  the  scope  of  the  present  T&E 
to  evaluate  the  statistical  algorithms  that  MDSS  uses  to  generate  alerts  for  disease 
outbreaks  and  trends.  The  MDSS  algorithms  have  been  patented  and  have  undergone 
scientific  review  as  part  of  the  patent  process.  The  developers  are  encouraged  to  present 
the  data  supporting  their  choice  of  algorithms  at  professional  conferences  and  in  articles 
submitted  to  peer-reviewed  journals. 

Second,  several  types  of  information  from  the  extended  use  survey  indicated  that  users 
were  strongly  in  favor  of  the  concept  behind  MDSS  but  did  not  believe  that  it  was  ready 
for  routine  use.  Users  agreed  that  the  quality  of  MDSS  information  was  better  than  that 
supplied  by  previous  reporting  methods,  but  overall,  they  were  undecided  as  to  whether  it 
helped  them  do  their  jobs  (questions  6  and  10,  Figure  6).  Users  also  reported  relatively 
weak  but  positive  ratings  when  asked  if  MDSS  was  easy  to  use  and  navigate  (questions  2 
and  9,  Figure  6).  The  extended  use  ratings  were  not  as  strongly  positive  as  those 
provided  immediately  following  training.  The  results  suggest  that  when  MDSS  was 
operated  in  a  training  environment  with  the  developer  present  in  a  planned 
demonstration,  the  product  received  higher  ratings  than  when  users  based  their  ratings  on 
usage  in  a  real-world  work  environment.  For  instance,  Figures  3  and  7  show 
respondents’  ratings  of  MDSS  features  such  as  the  alert  matrix  after  training  and  after 
extended  use,  respectively.  Responses  to  the  extended  use  survey  show  variability  across 
features  and  lower  ratings  for  the  alert  matrix  in  particular. 

User  comments  also  differed  after  training  and  extended  use.  The  most  prevalent 
posttraining  comments  by  users  were  that  MDSS  was  a  “good  system”  and  that  the  users 
would  “have  to  use  the  system  more,”  suggesting  that  it  was  appealing  for  them  to  leam. 
In  contrast,  after  extended  use,  respondents  commented  mostly  on  a  specific  positive 
aspect,  such  as  the  “valuable  graphics  tools  and  reports.”  They  also  isolated  concerns 
about  data  input  problems  and  the  design  of  the  system  (“difficult  to  use”,  “want  to  see 
diagnoses”;  Figures  4  and  8). 

The  key  feature  of  MDSS  is  real-time  automated  surveillance,  augmented  by 
user-controlled  “drill-down”  or  epidemiological  tools  to  investigate  disease  trends.  None 
of  the  users  appeared  to  integrate  the  automated  surveillance  feature  into  their  work 
routinely,  mostly  because  of  problems  with  data  input  to  MDSS.  However,  users  did 
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make  extensive  use  of  data  analysis  features  retrospectively  for  job  reporting 
requirements  and  disease  tracking  from  month  to  month.  Analysis  of  the  extended  use 
comments  was  consistent  with  the  findings  from  the  numerical  ratings.  The  most 
prevalent  comment  was  that  the  graphics  tools  and  reports  were  valuable,  followed  by 
negative  comments  on  data  input  problems  (Figure  8). 

One  regular  user  did  comment  that  the  MDSS  tool  was  the  first  source  to  detect  several 
tuberculosis  cases  prior  to  other  usual  sources.  This  is  the  only  evidence  of  primary 
detection  of  important  disease  occurrences  noted  in  this  study.  However,  the  same  user 
also  reported  that  MDSS  did  not  aid  in  the  detection  of  a  local  outbreak  of  Strep  A  at 
MCRD  San  Diego.  The  alert  matrix  for  this  user  had  not  been  functional  due  to  data 
input  problems,  limiting  its  utility  for  this  outbreak.  The  user  went  on  to  speculate  that 
MDSS  would  have  provided  alert  of  this  outbreak  no  sooner  or  later  than  current  methods 
of  reporting,  namely  telephone  calls  from  the  affected  clinics. 

Third,  a  number  of  modifications  were  recommended  to  improve  product  utility  or 
usability.  See  also  sections  5.4  and  6.0  in  this  report  on  Issues,  Solutions  and 
Recommendations : 

•  A  standard  operating  procedure  for  MDSS  could  be  developed  through 
discussion  with  current  users  and  presented  during  training.  This  could 
include  the  personnel  and  time  necessary  to  allow  optimal  usage  of  the 
system.  Several  primary  users  commented  that  substantial  person  power 
is  required  to  fully  utilize  and  evaluate  MDSS. 

•  Alerts  might  be  most  effective  when  triggered  by  clinically  functional 
syndrome  groups  rather  than  traditional  reporting,  such  as  DNBI  or  Major 
ICD-9  groupings.  The  Key  Symptoms  groupings  appear  to  be  a  strong 
step  in  this  direction.  Research  to  validate  the  utility  of  the  groupings 
would  be  essential.12 

•  ICD-9  groupings  were  not  functional  because  they  are  too  general  as 
currently  generated.12, 13  A  more  specific  list  of  top  10  ICD-9  codes  may 
be  more  useful  for  surveillance  and  reporting  purposes.  This  is  related  to 
the  above  point.  In  future  versions,  developers  should  carefully  refine 
special  groupings  like  this  and  conduct  an  evaluation  of  their  utility. 

•  Though  the  training  sessions  were  well  received,  follow-up  sessions 
should  be  conducted  where  possible  on  important  features.  Such  a 
feedback  mechanism  would  allow  developers  to  take  proactive  steps  to 
intervene  and  reinforce  important  features,  especially  to  relatively  new 
users.  During  the  present  study,  evidence  indicated  that  users  requested 
features  that  were  already  in  place  in  some  form,  such  as  Key  Symptoms 
and  CBR  reports  for  biochemical  terror  threats. 

•  Ultimately,  other  data  inputs  to  MDSS,  such  as  pharmacy  and  ER  data, 
might  be  valuable  for  comprehensive  surveillance.12 

•  A  geographical  mapping  function  to  show  the  distribution  of  locations  of 
disease  trends  and  outbreaks  could  be  useful.14 
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Finally,  As  MDSS  and  other  Web-based  applications  for  disease  surveillance  mature,  a 
substantial  number  of  research  reports  on  these  technologies  will  begin  to  appear  in  the 
literature.12'16  Therefore,  it  will  become  essential  to  provide  current  critical  review  of  the 
literature  to  optimize  the  development  and  evaluation  of  these  products. 

8.0  Study  Strengths  and  Limitations 

This  section  reviews  the  strengths  and  limits  of  the  present  study  design  and  its 
measurement  instruments. 

8.1  Converging  Evidence 

The  primary  strength  of  the  present  T&E  design  comes  from  the  ability  to 
evaluate  the  MDSS  product  based  on  converging  evidence  from  several  sources. 
Users  completed  surveys  after  training  and  again  several  months  later,  after  they 
had  had  an  opportunity  to  use  MDSS  as  a  routine  part  of  their  jobs.  The  T&E 
team  also  followed  up  on  responses  to  the  extended  use  surveys  to  interview 
primary  users.  Thus,  data  included  both  numerical  ratings  of  usability  and 
satisfaction  and  comments  by  users.  A  third  source  of  evidence  came  from  the 
controlled  validation  tests  of  specific  MDSS  functions. 

8.2  Scientific  Validity  of  Surveys 

The  present  results  are  primarily  descriptive  because  of  limited  sample  sizes 
available  and  therefore  trends  from  a  small  number  of  individuals  are  emphasized. 
However,  it  is  worth  noting  that  with  relatively  few  respondents,  each  individual 
was  tracked  more  carefully  and  they  appeared  more  motivated  to  fill  out  the 
survey.  The  extra  time  taken  by  many  respondents  to  write  comments  also  is 
consistent  with  their  careful  attention  to  the  survey  task. 

Samples  sizes  were  too  small  to  test  reliability  and  validity  of  the  current  surveys 
but  the  general  pattern  of  results  was  reasonable.  For  instance,  survey  ratings  of 
product  performance  were  most  favorable  under  controlled  conditions  during 
training  and  validation  testing.  These  ratings  were  less  favorable  when  the 
product  was  installed  at  MTFs  for  routine  use  with  ICD-9  data  input  problems  and 
a  busy  working  environment. 


8.3  Study  Limitations 

As  mentioned,  the  main  limitation  of  this  study  was  small  sample  sizes.  Three 
users  tested  this  product  regularly  on  the  job  for  2  to  3  months  and  therefore  had 
extensive  opportunity  to  use  the  product  and  were  able  to  provide  reasonable  and 
systematic  survey  responses.  However,  it  is  not  clear  how  the  results  from  these 
individuals  will  generalize  to  other  users.  Some  respondents  were  corpsmen  and 
reported  not  using  MDSS  regularly.  This  could  be  because  of  inexperience  with 
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surveillance  and  epidemiology,  or  because  their  jobs  did  not  require  surveillance, 
or  because  they  were  located  at  clinics  that  had  problems  with  access  to  MDSS. 
These  respondents  might  be  more  or  less  technically  oriented  with  regard  to 
software  applications  and  surveillance  than  other  medical  personnel. 

A  specific  limitation  of  the  present  T&E  was  related  to  the  timely  data  input 
problems,  which  prevented  complete  evaluation  of  the  system’s  critical  feature, 
disease  detection  and  alerting  through  DCD  analysis.  Since  outlying  clinics  often 
did  not  enter  data  in  near  real-time,  the  PMOs  did  not  trust  the  alert  matrix  reports 
and  did  not  use  this  feature  extensively.  This  was  not  an  MDSS  design  flaw,  but 
it  did  limit  the  T&E  effort  with  regard  to  how  the  alert  matrix  might  help  medical 
personnel  in  real-time  surveillance  as  a  routine  part  of  their  jobs. 
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Appendix  A 

MDSS  User  Training  Survey 
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MDSS  User  Training  Survey 

Name:  _  Location:  _ 

Date:  _  Instructor:  _ 

What  is  your  present  position  title  (e.g.,  preventive  medicine  officer)?  _ 

What  is  the  area  of  your  training  (e.g.,  epidemiology,  statistics)?  _ 

1.  User  training  provided  me  with  an  understanding  of  the  purpose  of  MDSS. 

□  Strongly  Agree  □  Agree  □  Undecided  □  Disagree  □  Strongly  Disagree  □  Not  Applicable 

Comments: 


2.  Now  that  the  training  has  been  completed,  I  understand  how  MDSS  supports  my 
job. 

□  Strongly  Agree  □  Agree  □  Undecided  □  Disagree  □  Strongly  Disagree  □  Not  Applicable 

Comments: 


3.  I  am  prepared  to  use  MDSS. 

□  Strongly  Agree  □  Agree  □  Undecided  □  Disagree  □  Strongly  Disagree  □  Not  Applicable 


Comments: 
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4.  There  was  sufficient  time  allocated  for  training  on  MDSS. 

□  Strongly  Agree  □  Agree  □  Undecided  □  Disagree  □  Strongly  Disagree  □  Not  Applicable 

Comments: 


5.  I  would  recommend  MDSS  to  my  command. 

□  Strongly  Agree  □  Agree  □  Undecided  □  Disagree  □  Strongly  Disagree  □  Not  Applicable 

Comments: 


6.  Please  rate  the  usefulness  of  each  of  these  MDSS  capabilities  on  a  scale  of  1-5. 

(1  =  not  at  all  useful;  2  =  not  useful;  3  =  undecided;  4  =  useful;  5  =  very  useful) 

_  MDSS  Alert  Matrix 

_  Trend  Analysis  Reports 

_  Medical  Threat  Assessment 

_  Patient  Records  Reports 

_  Data  Analysis  Capabilities 

_  "Drill-Down"  or  Investigation  Capabilities 

_  Ad  Hoc  Reports 

_  Other  (please  specify)  _ 

7.  Do  you  have  any  other  observations  or  comments  about  the  training  or  the  MDSS 
software? 
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MDSS  User  Survey 

Please  fill  out  this  brief  survey  (10-15  minutes)  as  completely  as  you  can.  Return  it  in  the 
addressed  envelope  provided.  This  work  is  sponsored  by  the  Office  of  Naval  Research  and  is 
being  conducted  by  Naval  Health  Research  Center  to  help  improve  the  Medical  Data 
Surveillance  System  (MDSS).  Your  responses  will  remain  confidential  and  your  privacy  will  be 
maintained  throughout  this  research. 

Questions  can  be  directed  to  Ralph  Burr,  M.A.,  at: 

Naval  Health  Research  Center 
P.O.  Box  85122 
San  Diego,  CA  92 186-5 122 
619-553-7760 
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Your  Name  _ 

Phone  _  E-mail  address  _ 

Date  _ 

Background 

Location  of  your  present  assignment: 

_ Naval  Medical  Center,  San  Diego 

_ Naval  Hospital  Camp  Pendleton 

_  Branch  Medical  Clinic  Chinhae  in  Korea 

_  U.S.  Naval  Hospital  in  Okinawa 

Other 


Gender:  _  Male  _  Female 

Active  Duty:  _  Yes  _  No 

Service: 

_  Navy 

_  Army 

_  Air  Force 

_  Marines 

What  is  your  present  position  title? 

_  GMO  (general  medical  officer) 

_  PMO  (preventive  medicine  officer) 

_  EHO  (environmental  health  officer) 

_  IDC  (independent  duty  corpsman) 

_  Nurse 

_  Physician’s  Assistant 
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Command  surgeon 

Task  force  surgeon 

CINC  surgeon 

Epidemiologist 

Task  Force  commander 

HAZMAT 

Other 


Which  area  best  describes  your  education  and/or  training  (choose  one)? 

_  Statistics 

_  Epidemiology 

_  Preventive  Medicine 

Other 


How  often  do  you  use  the  MDSS  system  presently  deployed  at  your  treatment  facility? 

_  I  use  it  daily 

_  I  use  it  a  few  times  a  week 

_  I  use  it  a  few  times  a  month 

_  I  rarely  use  this  system 


Please  describe  how  you  used  it.  What  features  did  you  use  in  your  job? 


How  much  experience  do  you  have  with  automated  surveillance  and  medical  data 
software  other  than  MDSS?  (months/years)  _ 

How  much  experience  do  you  have  working  aboard  a  deployed  ship? 
(months/years)  _ 
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Your  written  comments  will  assist  in  the  development  of  this 
technology  for  the  medical  support  of  the  armed  forces. 
Please  provide  comments  wherever  possible,  even  if  they  only 
include  a  few  key  words. 


MDSS  Information 

1 .  The  medical  threat  information  provided  through  MDSS  was  useful. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


2.  The  medical  threat  information  provided  through  MDSS  was  easy  to  use. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


3.  The  medical  threat  information  provided  through  MDSS  has  the  potential  to  be  used  for 
decision  making. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


4.  Medical  threat  information  from  MDSS  was  available  fast  enough  to  use  in  operational 
decision  making. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 
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5.  The  medical  threat  information  MDSS  provided  was  presented  in  a  useful  format. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


6.  The  quality  of  the  medical  threat  information  provided  by  MDSS  is  better  than  that  provided  by 
previous  reporting  methods.  (Please  specify  reporting  methods  previously  used.) 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


7.  I  want  MDSS  in  my  next  deployment. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


8.  The  MDSS  system  was  flexible  enough  to  meet  my  needs.  I  could  set  it  up  to  do  what  I  wanted 
it  to  do  efficiently. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 
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9.  The  following  is  a  list  of  MDSS  component  functions  or  programs  that  you  may  have  used  in 
the  last  few  months.  Please  indicate  your  level  of  satisfaction.  Please  use  the  space  below  each 
item  to  comment  why  you  rated  each  item  the  way  you  did. 


a)  Initial  Threat  Overview  page/ Alert  Matrix 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


b)  Disease  and  Non-Battle  Injury  (DNBI)  reports 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


c)  Dynamic  Change-Point  Detection  (DCD)  capabilities 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


d)  Pivot  tables 

□  Very  Satisfied  □  Satisfied  □  Undecided  □Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


e)  Epidemiological  Tools/Drill-Down  Investigation  Capabilities 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


f)  Population  at  Risk  (PAR)  Tables 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


g)  Calendar  function  for  reduced  staff  days 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 
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h)  Graphs 

□  Very  Satisfied  □  Satisfied  □  Undecided  □  Dissatisfied  □  Very  Dissatisfied  □  Not  Observed 


i)  Other: 


General  Comments: 


10.  It  was  easy  to  navigate  the  MDSS  system. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


1 1 .  MDSS  helped  me  do  my  job. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  Not  Observed 

Please  Explain:  _ 


12.  The  most  useful  thing  about  MDSS  for  my  job  was 


13.  The  least  useful  thing  about  MDSS  for  my  job  was 


14.  If  I  could  add  one  thing  to  MDSS,  it  would  be 


B-7 


Final  Report 


Test  &  Evaluation  During  Extended  MDSS  Deployment 


15.  The  introductory  training  provided  by  the  product  developers  prepared  me  to  use  MDSS. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  I  did  not  receive  any  formal  training  on  this  product 

Please  Explain:  _ 


16.  The  CD  I  received  after  training  containing  the  lessons  and  user  guides  helped  me  use  MDSS. 

□  Strongly  Agree  □  Agree  □  Neither  Agree  Nor  Disagree  □  Disagree  □  Strongly  Disagree 

□  I  did  not  use  the  reference  CD 

Please  Explain:  _ 
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Task  Usability  Issue/Comments 


User  Role:  Administrator 

Login  as  MDSS,  password:  MDSS _ 

Enter  the  Administrator  Control  Panel. _ 

View  current  database  records. _ 

Grant  user  “Bohannan”  administrative  privileges. _ 

Create  a  new  group  named  NHRC  and  grant  access  to  all  options. 

View  population  at  risk  (PAR). _ 

Modify  Entity  2BN4MAR  Longitude  to  26. _ 

Add  an  entity  to  the  PAR  administration  named  “Null  Unit  B.” 


Assign  a  reduced  staff  weekend  for  Null  MTF. 
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Task 


Log  off  MDSS 


User  Role:  Surveillance 


Log  onto  MDSS  with  username:  NHRC  and  Password:  NHRC 


Usability  Issue/Comments 


View  CBR  Report,  All  MTFs,  All  Units,  All  visit  types,  weekly  report,  2002-09-29 
end  date. 


View  Weekly  rate  per  thousand  for  Fever  cateeo 


a 


View  Patient  information  for  dates  2002-09-22  to  2002-09-29,  full  staff  days. 


Generate  a  Dynamic  Change-Point  Detection  Analysis  for  DNBI  category: 
Respiratory,  from  09-22  to  09-29,  all  visits,  all  units,  all  MTFs. 


Run  a  (Rate)  Background  Alert  for  MTF  Tri-Care  Outpatient  San  Diego  2,  Major 
ICD9. 


Use  the  “drill-down”  function  to  view  the  respiratory  illnesses  on  high  alert  for 
9/25/2002. 


Run  a  DNBI  Report  for  all  MTFs,  All  units,  All  visits,  Monthly  report  for  2002-09- 
29  end  date  using  calculated  PAR  values. 


View  weekly  incident  rates  for  Respiratory  catego 


a 
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Task 


View  DNBI  Report:  Monthly,  category  All  Other,  Medical/Surgical,  all  Units,  all 
MTFs,  Initial  visits,  2002-09-29  end  date. 


N  Usability  Issue/Comments 


View  Reportable  Conditions  Summary  Report  for  all  Units,  all  MTFs,  Initial  visits, 
Weekly  report,  and  09-29-2002  as  the  end  date. 


Run  Major  ICD-9  Report  for  all  Units,  all  MTFs,  Initial  visits,  daily  report  and  09- 
29-2002  end  date. 


Run  2x2  Contingency  Table  (Time  Interval)  for  all  Units,  CBR  category: 
“breathless,  cough,  sore  throat,  etc”,  for  All  Patients,  Initial  visits.  Start/End  dates 
1:2002-09-15,2002-09-21 


Start/End  dates  2:  2002-09-22,  2002-09-29 


Run  Background  Alert  (Count)  for  MTF:  NBMC  NAS  North  Island,  summary 
reportable  conditions 


View  “Occupational  Exposure  to  Blood  Borne  Pathogens”  graphs  under  the  link 
with  the  same  title. 


Access  the  Ill  Defined  Conditions  category  with  alert  status  on  MTF  TriCare 
Outrpatient  San  Diego  2  on  the  Total  Count  Alert  Matrix. 


View  detail  for  Abdominal  Pain  in  the  Full  Staff  Days  table. 


View  patient  information  for  2002-09-22  to  2002-09-29. 
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Task  Y  N  Usability  Issue/Comments 

Log  off  MDSS 


User  Role:  Non-medical _ 

Log  onto  MDSS  with  username:  Bohannan  and  Password:  bb 


Generate  a  Reportable  Conditions  Summary  Report  on  all  MTFs,  all  Military  Units, 
follow-up  visits,  for  end  date  2002-09-29 


View  Weekly  Incidence  for  Occupational  exposure  to  Blood  Borne  Pathogens. 


View  Major  ICD9  Report  for  MBNC  Coronado,  All  Units,  All  Visits,  2002-09-29 
end  date,  Weekly  Report. 


View  weekly  Incidence  -  Ill  Defined  conditions. 


Log  off 
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